Real-Time Dense Stereo Matching With ELAS on FPGA Accelerated Embedded Devices

For many applications in low-power real-time robotics, stereo cameras are the sensors of choice for depth perception as they are typically cheaper and more versatile than their active counterparts. Their biggest drawback, however, is that they do not directly sense depth maps; instead, these must be estimated through data-intensive processes. Therefore, appropriate algorithm selection plays an important role in achieving the desired performance characteristics. Motivated by applications in space and mobile robotics, we implement and evaluate a FPGA-accelerated adaptation of the ELAS algorithm. Despite offering one of the best trade-offs between efficiency and accuracy, ELAS has only been shown to run at 1.5-3 fps on a high-end CPU. Our system preserves all intriguing properties of the original algorithm, such as the slanted plane priors, but can achieve a frame rate of 47fps whilst consuming under 4W of power. Unlike previous FPGA based designs, we take advantage of both components on the CPU/FPGA System-on-Chip to showcase the strategy necessary to accelerate more complex and computationally diverse algorithms for such low power, real-time systems.Comment: 8 pages, 7 figures, 2 table

R3^3SGM: Real-time Raster-Respecting Semi-Global Matching for Power-Constrained Systems

Stereo depth estimation is used for many computer vision applications. Though many popular methods strive solely for depth quality, for real-time mobile applications (e.g. prosthetic glasses or micro-UAVs), speed and power efficiency are equally, if not more, important. Many real-world systems rely on Semi-Global Matching (SGM) to achieve a good accuracy vs. speed balance, but power efficiency is hard to achieve with conventional hardware, making the use of embedded devices such as FPGAs attractive for low-power applications. However, the full SGM algorithm is ill-suited to deployment on FPGAs, and so most FPGA variants of it are partial, at the expense of accuracy. In a non-FPGA context, the accuracy of SGM has been improved by More Global Matching (MGM), which also helps tackle the streaking artifacts that afflict SGM. In this paper, we propose a novel, resource-efficient method that is inspired by MGM's techniques for improving depth quality, but which can be implemented to run in real time on a low-power FPGA. Through evaluation on multiple datasets (KITTI and Middlebury), we show that in comparison to other real-time capable stereo approaches, we can achieve a state-of-the-art balance between accuracy, power efficiency and speed, making our approach highly desirable for use in real-time systems with limited power.Comment: Accepted in FPT 2018 as Oral presentation, 8 pages, 6 figures, 4 table

Balancing accuracy, speed and power in the computation of depth with stereo cameras

Many real-world applications in robotics and computer vision rely on the ability to acquire reliable 3-D information. Often, these systems also have strict requirements on the speed, latency, power consumption, form factor, usability and cost with which this capability is achieved. Although stereo cameras offer a cost-effective and versatile means of acquiring such depth information, using them involves computationally intensive processing over large amounts of data. At the same time, due to physical limitations on power consumption, the operating frequency of traditional single-core processors has plateaued. This has, consequently, spurred the adoption and development of multi-core and application-specific devices instead. As a result, algorithms can no longer be expected to simply improve in speed and efficiency with newer generations of processors. Instead, to achieve truly low-power and real-time systems requires algorithms that are purposely designed and optimized to account for the strengths and limitations of the underlying hardware. Through efficient pipelining, extensive parallelism and conservative allocation of hardware resources, Field Programmable Gate Arrays (FPGA) are highly suited to fast, low-latency and power-efficient algorithm implementations. In this thesis, the capabilities of these devices are exploited for fast and power-efficient stereo image processing. Algorithms are investigated, developed and described for both strictly FPGA-only systems as well as heterogeneous CPU/FPGA System-on-Chip (SoC) platforms. The contributions of this thesis lie not only in the new approaches to low-power and real-time depth perception from stereo images that it proposes, but also in the insight that it provides regarding how to best utilize the FPGA-based devices for maximum performance. Simple, but highly parallel and predictable algorithms are shown to be great candidates for pure FPGA implementations, but more complex processing pipelines benefit more from the mixed computational strengths of the CPU-FPGA SoCs. The thesis also attempts to address the most significant disadvantage of using FPGAs, namely long development times involving highly specialized languages, by demonstrating how high-performing implementations can be obtained with the efficient use of high-level synthesis instead